11L GEPA + 20k Steps + Pure Int6 + Legal TTT → 1.0983 BPP

Non-record unlimited-compute submission. Breaks 1.10 BPP with legal score-first TTT.

Key Numbers

Scaling Table

Research Contributions (5 transferable findings)

1. Warmdown is a first-class variable — The model plateaus at ~1.216 BPP during late peak-LR (steps 7k–12k, ~2 BPB/kstep), then warmdown delivers −0.101 BPB at 12.6 BPB/kstep — roughly 6× the late-plateau rate. Warmdown isn't cleanup; it's where most remaining gain originates once the plateau sets in.

2. Better-trained models compress smaller — 20k-step model → 14.29 MB (smallest artifact), despite identical architecture and quantization. Optimization quality improves weight compressibility, not just float loss.

3. SGD >> AdamW for legal TTT (controlled comparison) — On the same 5.2k-step, 24.6M-param base model: SGD+momentum delivers 2.4× the TTT gain of AdamW (−0.017 vs −0.007 float→final). Adam's moments can't converge in ~30 steps/chunk. Separately, the 20k GEPA model's −0.044 TTT gain is measured from a different baseline (quant→final) and different architecture, so should not be directly compared.

4. Freezing early layers is active regularization — Freezing 2 of 11 blocks (~18% of depth) during TTT isn't just catastrophic-forgetting defense. Early layers hold generic features; later layers are the better adaptation surface. Even though freezing removes parameters, the model adapts better.

5. After the right TTT family, invest in the base model — TTT's share of total gain over naive baseline shrinks from 22% (5.2k-step base) to 13% (20k-step base). The big jump came from choosing the right TTT regime (SGD + freeze + multi-epoch). After that, base model quality delivers more BPB per unit of effort than TTT micro-tuning.

What Transfers to Record Track

✅ Warmdown emphasis (≥40% of total steps)
✅ GPTQ-lite / pure int6
✅ SGD-based legal TTT (2.4× gain over AdamW, validated on same base)
✅ Freeze-early-blocks as TTT regularization

⚠️ Less transferable: very long training curves, large eval-time TTT budgets (10–30 epochs → 2000–3600s eval)

Open Frontiers

The local TTT recipe appears mostly saturated. Next questions are structural: stream vs. document-based adaptation, self-distillation at test time, quantization-aware TTT, and base-training scaling laws under fixed 16 MB budget.

Full analysis with all tables and derivations in README.md.

Prior Non-Record Submissions

• PR Non-record: 11L GEPA + 12k Steps + Pure Int6 + Legal TTT (val_bpb=1.1079) #612: 12k steps → 1.1079 BPP
• PR Non-Record: BPB 1.1334 — 7000-Step Training + Mixed Int6/Int8 Quantization + Legal TTT #598: 7k steps + Mixed Int6/Int8 → 1.1334 BPP

Acknowledgments

Builds on techniques from: @signalrush (PR #414, GPTQ-lite/EMA), @jfprincz (PRs #287/#315, XSA/Partial RoPE/LN Scale), @unnir (PR #265, Efficient XSA), raahilshah (PR #162, SmearGate/BigramHash), @aruniyer (PR #86, Int6 QAT), samacqua (LoRA TTT), @abaybektursun (PR #549, LeakyReLU²), and the OpenAI baseline.